feat: pi-computer-use-style desktop computer use for all providers - #77
Merged
Conversation
…on layer Design doc in docs/computer-use.md. Permissions API (check/request/ open pane/relaunch) is the seam shared by the MCP backend and the upcoming Settings pages. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All three provider paths (Claude --mcp-config, Codex -c mcp_servers, ACP session/new mcpServers) now carry the tcode_computer_use server, gated on the new global computer_use.enabled setting. Tool bodies are permission-checked stubs behind a single dispatch seam. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…start continuity Two new settings pages with live permission rows (grant / recheck / relaunch), image-mode + observe-only knobs, browser enable / home URL / JS-evaluate gates on the preview panel, and a relaunch.json marker so a TCC-triggered restart reopens the recorded session and settings page. Verified against HEAD in an isolated worktree (fmt, clippy -D warnings, workspace tests, tcode build) while the backend crate is mid-rewrite. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…put, state store Full engine behind the tool seam: bounded immutable state store with per-root epochs, folded outline rendering + ranked search, AX tree observation, CGEvent input synthesis with key-chord parsing, window screenshot capture, honest per-step act_ui outcomes with expect postconditions, and a cu-smoke binary for scripted VM verification. Non-macOS targets build a structured stub. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The unicode keyboard path dropped its payload: the key-up event never carried the UTF-16 string and stale modifier flags leaked in, so apps ignored the pair. Verified end to end in the tart VM (TextEdit smoke exits 0, expect=verified). cu-smoke gains hidden --find-roots / --observe / --search / --act / --screenshot subcommands that reuse the production tool layer for scripted debugging. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
In-process SCScreenshotManager capture (exact CGWindowID filter, point size x backing scale, 5s timeout, ImageIO PNG) replaces the screencapture subprocess; the CLI path remains as a runtime fallback. Verified in the tart VM: SCK path confirmed via temporary marker, valid 1206x1010 window PNG, full TextEdit smoke exit 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…need preview_screenshot now uses WKWebView takeSnapshot via wry's macOS extension (PNG through ImageIO, 5s timeout) instead of capturing the on-screen region with screencapture. The embedded browser therefore needs no TCC permission at all: the Screen Recording row and restart banner leave the Browser settings page, and the screen-region capture path is removed from services/runtime. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Tryanks
added a commit
that referenced
this pull request
Sep 7, 2026
Computer-use and preview tool descriptions were single sentences since their introduction (#77, 092a12c). The state-lifecycle, permission, observe-only and browser-boundary rules lived only in docs/computer-use.md, and the preview_status -> preview_open workflow lived only in the Codex developer instructions, so Claude Code sessions never saw either; runtime errors were the model's only teacher. Move those rules into the #[tool] descriptions and the server instructions. No behavior change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements pi-computer-use-style desktop computer use in tcode, available to every provider (Claude Code, Codex, and ACP agents with
mcpCapabilities.http) via a new in-processtcode_computer_useMCP server. Design doc:docs/computer-use.md.Computer-use engine (
crates/computer-use-mcp)find_roots,observe_ui,search_ui,expand_ui,inspect_ui,act_ui,read_text,wait_for— state-scoped refs (@rN/@eN), bounded immutable observation store with per-root epochs, folded outlines with progressive disclosure, honest per-stepworked/didnt/unknownoutcomes withexpectpostconditions, bounded output with@ocontinuations.screencapturewindow capture attached per image-mode. Other platforms build a structured stub, keeping the three-platform CI matrix green.cu-smokebinary:--permissionsJSON status plus a scripted TextEdit end-to-end smoke, and hidden debug subcommands that drive the production tool layer for VM testing.Settings, permissions UX, restart continuity
relaunch.jsonmarker written before any grant/relaunch; on startup tcode reopens the recorded session and settings page and rechecks permissions — surviving macOS's own "quit & reopen" after a Screen Recording grant. In-app Relaunch tcode button on the restart banner.settings.jsonfiles load unchanged.Verification
cargo fmt/cargo clippy --workspace -D warnings/ full workspace tests green locally (macOS).find_roots → observe_ui → act_ui type → wait_for, exit 0); settings pages, grant deep-links, recheck, restart banner, relaunch continuity, and MCP registration all verified on-screen with screenshot evidence.type_textevents dropped their payload (missing UTF-16 string on key-up + stale modifier flags).🤖 Generated with Claude Code